Image processing device, image processing method, and vehicle

ABSTRACT

Provided is an image processing device that includes a first generation unit that projects a first image on a plurality of planes on a virtual spherical surface according to projection angles obtained by dividing a viewing angle at time of capturing the first image to generate the plurality of basis planar images. The image processing device further includes a second generation unit that projects a second image on a plurality of planes on a virtual spherical surface according to projection angles obtained by dividing a viewing angle at time of capturing the second image having an imaging range overlapping with the first image to generate the plurality of reference planar images, and a plurality of stereo image processing units that performs stereo image processing using a corresponding image pair of the plurality of generated basis planar images and reference planar images to generate distance information indicating a distance to an object on the basis image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2017/019491 filed on May 25, 2017, which claimspriority benefit of Japanese Patent Application No. JP 2016-114387 filedin the Japan Patent Office on Jun. 8, 2016. Each of the above-referencedapplications is hereby incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present technology relates to an image processing device, an imageprocessing method, and a vehicle, and in particular to an imageprocessing device, an image processing method, and a vehicle suitablefor a case of measuring a distance to an object using the principle oftriangulation, using an image pair simultaneously imaged by two (ormore) cameras arranged in a line.

BACKGROUND ART

To recognize an object in a three-dimensional space, there is atechnology of measuring a distance to the object. As measurement of thedistance to an object, stereo image processing is known (for example,see Patent Document 1), in which the object is simultaneously imaged bytwo (or more) cameras having a parallax by being separately arranged bya predetermined base line length, and the distance to the object ismeasured using the principle of triangulation, using a pair of a basisimage and a reference image (hereinafter referred to as stereo imagepair) obtained as a result of the imaging.

In the stereo image processing, it is important to accurately detect thepositions of corresponding points (objects) in the stereo image pair.Note that, in an ideal state, the corresponding points (objects) in thestereo image pair exist in a direction parallel to a line connectingcenters of the two cameras. Thus, it is sufficient to search only thatdirection. Here, the ideal state means a case in which the two camerascan project a three-dimensional space to be imaged on an ideal planewithout distortion with respect to an arbitrary direction.

By the way, to recognize an object existing in a wider range in thethree-dimensional space, a wide-angle camera to which a fisheye lens orthe like is attached and capable of imaging a stereo image pair at awide viewing angle is sometimes used.

Here, the wide-angle camera is defined as a camera capable of imaging anobject at a wider viewing angle than a normal camera, and provided witha wide-angle lens or a fisheye lens with a focal length of 35 mm orless, particularly, 28 mm or less in 35-mm conversion, for example.Furthermore, the wide-angle camera includes a camera capable of imagingan object at the viewing angle of 120 degrees or more, particularly 150degrees or more. Hereinafter, an image imaged by the wide-angle camerais referred to as a wide-angle image.

In a case where a stereo image pair is imaged by the wide-angle camera,the stereo image pair is greatly apart from the above ideal state, andgreater distortion occurs toward a peripheral portion in each of theobtained two wide-angle images. Therefore, detecting correspondingpoints on the wide-angle image pair becomes difficult. Therefore, amethod of aberration-correcting such distortion and detectingcorresponding points on a planar image pair obtained as a result of thecorrection has been proposed (for example, see Patent Document 2).

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    05-114099-   Patent Document 2: Japanese Patent Application Laid-Open No.    2001-235819

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

However, the aberration correction of the distortion described above isperformed only for a part of the imaged wide-angle image, and in a casewhere an object for which the distance is to be measured is not presentin the corrected area (in other words, a case in which the object is notpresent on the planar image), the positions of the corresponding points(objects) on the stereo image pair cannot be detected.

The present technology has been made in view of such a situation, and anobject of the present invention is to measure the distance to an objectexisting in a wider range, using a stereo image pair including imageswith a wide viewing angle imaged by a wide-angle camera.

Solutions to Problems

An image processing device according to a first aspect of the presenttechnology includes a first generation unit configured to acquire afirst image, and project the first image on a plurality of planes on avirtual spherical surface according to projection angles obtained bydividing a viewing angle at time of capturing the first image togenerate the plurality of basis planar images, a second generation unitconfigured to acquire a second image including an area where an imagingrange overlaps with an imaging range of the first image, and project thesecond image on a plurality of planes on a virtual spherical surfaceaccording to projection angles obtained by dividing a viewing angle attime of capturing the second image to generate the plurality ofreference planar images, and a plurality of stereo image processingunits configured to perform stereo image processing using acorresponding image pair of the plurality of generated basis planarimages and the plurality of generated reference planar images togenerate distance information indicating a distance to an object on thebasis image.

At least one of the first image or the second image can be an imageimaged by a wide-angle camera.

The second generation unit can generate the plurality of referenceplanar images provided with a margin with respect to the plurality ofbasis planar images generated by the first generation unit.

A width of the margin can be determined on the basis of a base linelength between a first imaging unit that images the first image and asecond imaging unit that images the second image.

An arranging direction of the plurality of basis planar images and theplurality of reference planar images can be orthogonal to a direction ofa base line length between a first imaging unit that images the firstimage and a second imaging unit that images the second image.

An arranging direction of the plurality of basis planar images and theplurality of reference planar images can be orthogonal to a searchdirection of a corresponding point in the stereo image processing.

The image processing device according to the first aspect of the presenttechnology can include a distance information integration unitconfigured to integrate the plurality of pieces of generated distanceinformation.

The distance information integration unit can convert a coordinatesystem of the plurality of pieces of generated distance information.

The image processing device according to the first aspect of the presenttechnology can further include a first imaging unit configured to imagethe first image, and a second imaging unit configured to image thesecond image.

At least one of the first imaging unit or the second imaging unit caninclude a wide-angle camera.

The first imaging unit and the second imaging unit can be arranged sideby side in a horizontal direction.

The first imaging unit and the second imaging unit can be arranged upand down in a vertical direction.

An image processing method according to the first aspect of the presenttechnology includes, in the image processing method of an imageprocessing device, by the image processing device, a first generationstep of acquiring a first image, and projecting the first image on aplurality of planes on a virtual spherical surface according toprojection angles obtained by dividing a viewing angle at time ofcapturing the first image to generate the plurality of basis planarimages, a second generation step of acquiring a second image includingan area where an imaging range overlaps with an imaging range of thefirst image, and projecting the second image on a plurality of planes ona virtual spherical surface according to projection angles obtained bydividing a viewing angle at time of capturing the second image togenerate the plurality of reference planar images, and a plurality ofstereo image processing steps of performing stereo image processingusing a corresponding image pair of the plurality of generated basisplanar images and the plurality of generated reference planar images togenerate distance information indicating a distance to an object on thebasis image.

In the first aspect of the present technology, the first image isacquired, the first image is projected on a plurality of planes on avirtual spherical surface according to projection angles obtained bydividing a viewing angle at time of capturing the first image togenerate a plurality of basis planar images, the second image includingan area where imaging range overlaps with an imaging range of the firstimage is acquired, the second image is projected on a plurality ofplanes on a virtual spherical surface according to projection anglesobtained by dividing a viewing angle at time of capturing the secondimage to generate a plurality of reference planar images, and the stereoimage processing using a corresponding image pair of the plurality ofgenerated basis planar images and the plurality of generated referenceplanar images is performed to generate distance information indicating adistance to an object on the basis image.

A vehicle according to a second aspect of the present technologyincludes a first imaging unit configured to image a first image, asecond imaging unit configured to image second image including an areawhere an imaging range overlaps with an imaging range of the firstimage, a first generation unit configured to acquire the first image,and project the first image on a plurality of planes on a virtualspherical surface according to projection angles obtained by dividing aviewing angle at time of capturing the first image to generate theplurality of basis planar image, a second generation unit configured toacquire the second image, and project the second image on a plurality ofplanes on a virtual spherical surface according to projection anglesobtained by dividing a viewing angle at time of capturing the secondimage to generate the plurality of reference planar images, and aplurality of stereo image processing units configured to perform stereoimage processing using a corresponding image pair of the plurality ofgenerated basis planar images and the plurality of generated referenceplanar images to generate distance information indicating a distance toan object on the basis image.

In the second aspect of the present technology, the first image isimaged, the second image including an area where an imaging rangeoverlaps with an imaging range of the first image is imaged, the firstimage is projected on a plurality of planes on a virtual sphericalsurface according to projection angles obtained by dividing a viewingangle at time of capturing the first image to generate a plurality ofbasis planar image, the second image is projected on a plurality ofplanes on a virtual spherical surface according to projection anglesobtained by dividing a viewing angle at time of capturing the secondimage to generate a plurality of reference planar images, and the stereoimage processing using a corresponding image pair of the plurality ofgenerated basis planar images and the plurality of generated referenceplanar images is performed to generate distance information indicating adistance to an object on the basis image.

Effects of the Invention

According to the first and second aspects of the present technology, thedistance to an object existing in a wider range can be measured.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a relationship between a wide-angleimage and a planar image.

FIG. 2 is a diagram illustrating a relationship between a virtualspherical surface and a planar image.

FIG. 3 is a diagram illustrating a relationship between a wide-angleimage and a plurality of planar images serving as basis images.

FIG. 4 is a block diagram illustrating a configuration example of animage processing device to which the present technology is applied.

FIG. 5 is a diagram illustrating a relationship between a wide-angleimage and a plurality of planar images serving as reference images.

FIG. 6 is a diagram for describing an integration example of distanceinformation.

FIG. 7 is a diagram for describing an integration example of thedistance information.

FIG. 8 is a diagram for describing an integration example of thedistance information.

FIG. 9 is a flowchart for describing distance measurement processing byan image processing device.

FIG. 10 is a diagram illustrating an example of dividing a planar imagein a strip manner.

FIG. 11 is a diagram illustrating an arrangement example of a firstimaging unit and a second imaging unit in FIG. 10.

FIG. 12 is a block diagram illustrating a configuration example of ageneral-purpose computer.

FIG. 13 is a diagram illustrating an arrangement example of stereocameras in a vehicle.

FIG. 14 is a diagram illustrating an arrangement example of a firstimaging unit and a second imaging unit configuring a stereo camera in avehicle.

FIG. 15 is a diagram illustrating an arrangement example of the firstimaging unit and the second imaging unit configuring the stereo camerain a vehicle.

FIG. 16 is a diagram illustrating an arrangement example of the firstimaging unit and the second imaging unit configuring the stereo camerain a vehicle.

FIGS. 17A, 17B, and 17C are diagrams illustrating an arrangement exampleof the first imaging unit and the second imaging unit configuring thestereo camera in a vehicle.

FIGS. 18A and 18B are diagrams illustrating an arrangement example ofthe first imaging unit and the second imaging unit configuring thestereo camera in a vehicle.

FIG. 19 is a block diagram illustrating an example of a schematicconfiguration of a vehicle control system.

FIG. 20 is an explanatory diagram illustrating an example ofinstallation positions of a vehicle exterior information detection unitand imaging units.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, best modes for implementing the present technology(hereinafter referred to as embodiments) will be described in detailwith reference to the drawings.

<Relationship Between Wide-Angle Image Imaged by Wide-Angle Camera andPlanar Image>

FIG. 1 illustrates a relationship between a wide-angle image captured bya wide-angle camera and a planar image obtained by aberration-correctionfor the wide-angle image. Note that FIG. 1 illustrates only one of abasis image and a reference image used for stereo image processing.

In a wide-angle image W imaged by the wide-angle camera, a state inwhich a three-dimensional space in an imaging direction is projectedonto a virtual spherical surface S is displayed on the image, and largedistortion has occurred. In this state, detection of a correspondingpoint from a paired wide-angle image W is difficult. Therefore, byprojecting the wide-angle image W onto a plane on the virtual sphericalsurface S, a planar image P is generated.

Note that, in generating the planar image P, polar coordinate conversionprocessing is required in a peripheral portion of the wide-angle imageW, which is to be used for distance measurement at a subsequent stage.

FIG. 2 schematically two-dimensionally illustrates a relationshipbetween the virtual curved surface S and the planar image P. Note thatFIG. 2 illustrates only one of the basis image and the reference imageused for the stereo image processing.

The planar image P is generated by setting a projection angle θ withrespect to the virtual spherical surface S and projecting the wide-angleimage W. The width w of the generated planar image P is expressed usingthe projection angle θ as described in the following equation (1).w=2R·tan(θ/2)  (1)

Here, R is a radius of the virtual spherical surface S. θ is a value ofaround 0° to 180° (because there are some fisheye lenses having a wideviewing angle of 180° or more).

Note that, in a case where the projection angle θ is 180 degrees, thewidth w of the equation (1) diverges to infinity. In other words, in thecase of the wide-angle image W imaged at the viewing angle of 180degrees or more, the entire image cannot be projected onto one planarimage P. Therefore, in the present technology, a single wide-angle imageW is projected onto a plurality of planar images P, that is, a pluralityof planar images P is generated from one wide-angle image W. Thus, evenif an object of which the distance is to be measured is in a peripheralarea in the wide-angle image W, the distance to the object can bemeasured.

FIG. 3 illustrates a case in which one wide-angle image W is projectedonto three planar images P. Note that FIG. 3 illustrates only the basisimage, of the basis image and the reference image used for the stereoimage processing.

As illustrated in FIG. 3, a planar image P11 is generated by projectinga portion of an angle θ11 of the viewing angle of the wide-angle imageW. Similarly, a planar image P12 is generated by projecting a portion ofan angle θ12 of the viewing angle of the wide-angle image W. A planarimage P13 is generated by projecting a portion of an angle θ13 of theviewing angle of the wide-angle image W. Note that the projection angles611, 612, and 613 need to be set to include the entire viewing angle ofthe wide-angle image W.

For example, in a case where the viewing angle of the wide-angle image Wis 180 degrees, values of the projection angles θ11, θ12, and θ13 may becommon 60 degrees. Note that the values of the projection angles θ11,θ12, and θ13 do not have to be common. For example, the projection angleθ 12 may be widened, and the other projection angles θ11 and θ13 may benarrowed. Moreover, the values of the projection angles θ11, θ12, andθ13 may be made variable and may be changed according to the scene to beimaged. With the setting, the planar image P in a specific direction(for example, a direction in which presence of an object is estimated)can be widened.

Note that FIG. 3 illustrates the case in which the viewing angle of thewide-angle image W is divided into three angles. However, for example,the viewing angle of the wide-angle image W may be divided into twoangles to generate two planar images P from one wide-angle image W, orthe viewing angle of the wide-angle image W may be divided into four ormore angles to generate four or more planar images P from one wide-angleimage W. If processing of detecting corresponding points of each set ofthe basis image and the reference image is performed in parallel,detection of the corresponding points of the entire area of thewide-angle image W can be more promptly executed in the case where thenumber of the planar images P to be generated is increased.

<Configuration Example of Image Processing Device According to PresentEmbodiment>

FIG. 4 illustrates a configuration example of the image processingdevice that is the present embodiment.

An image processing device 10 executes stereo image processing using astereo image pair captured by two cameras (a first imaging unit 11 and asecond imaging unit 12), and is expected to be mounted on a vehicle suchas an automobile, for example. In the case where the image processingdevice 10 is mounted on an automobile, the distance to an object(another vehicle, a person, a line on a road, or the like) existingaround the automobile can be measured. A measurement result thereof isused, for example, to realize functions such as approach warning,collision avoidance brake, lane change warning, and automatic steering.

The image processing device 10 includes a first correction unit 13, asecond correction unit 14, a first planar image pair processing unit 15,a second planar image pair processing unit 16, a third planar image pairprocessing unit 17, a distance information integration unit 18, and adistance information analysis unit 19.

The images respectively imaged by the first imaging unit 11 and thesecond imaging unit 12 are supplied to the image processing device 10.Hereinafter, it is assumed that the image imaged by the first imagingunit 11 is used as the basis image and the image imaged by the secondimaging unit 12 is used as the reference image.

Here, the first imaging unit 11 and the second imaging unit 12 arewide-angle cameras having an equal focal length and are arranged toimage substantially the same imaging range with a predetermined baseline length away from each other. Here, the predetermined base linelength is assumed to be about 5 cm in consideration of installation ofthe first imaging unit 11 and the second imaging unit 12 on doormirrors, especially in the case of on-vehicle mounting, for example.However, a base line length of 20 cm or the like can be set by securing5 cm or more, for example, 10 cm of a base line length using alarge-sized door mirror or providing a camera on a vehicle body.

The arranging direction of the first imaging unit 11 and the secondimaging unit 12 is typically a lateral direction (horizontal linedirection) but the arranging direction may be a vertical direction(vertical direction). In the case where the first imaging unit 11 andthe second imaging unit 12 are arranged in the vertical direction, theparallax between the first imaging unit 11 and the second imaging unit12 does not cross boundaries of a plurality of generated planar images(described below), and thus there is an advantage of easily detectingthe corresponding points. Furthermore, in the case of on-vehiclemounting, the parallax in the vertical direction is sometimes moreimportant than that in the lateral direction.

The first imaging unit 11 images the imaging range and outputs aresultant wide-angle image W1 to the first correction unit 13. Thesecond imaging unit 12 images the imaging range at the same imagingtiming as the first imaging unit 11 and outputs a resultant wide-angleimage W2 to the second correction unit 14. In this case, the imageprocessing device 10 can measure the distance to any object existinganywhere in the imaging range.

Note that, even in a case of adopting a standard camera in which a lenswith a standard viewing angle (a focal length is about 50 mm in 35-mmconversion) is mounted for the first imaging unit 11 and the secondimaging unit 12, instead of the wide-angle camera, the presenttechnology is also applicable.

Furthermore, the first imaging unit 11 and the second imaging unit 12may have different imaging directions as long as there is an overlappingarea in the respective imaging ranges. In this case, it is possible tomeasure the distance to an object existing in the area where the imagingranges of the first imaging unit 11 and the second imaging unit 12overlap with each other. Although it is not possible to measure thedistance to an object existing in an area where the imaging ranges ofthe first imaging unit 11 and the second imaging unit 12 do not overlapwith each other, the areas in the wide-angle images W1 and W2 can beused for a wide range of monitoring, and the like.

Moreover, the focal length of the attached lenses of the first imagingunit 11 and the second imaging unit 12 may be different. For example, awide-angle lens may be mounted on one of the first imaging unit 11 andthe second imaging unit 12 and a telephoto lens or the like with alonger focal length than the wide-angle lens, that is, a narrow viewingangle and higher resolution may be attached to the other imaging unit.Note that the first imaging unit 11 and the second imaging unit 12 areinstalled to cause the overlapping area in the imaging ranges of thefirst imaging unit 11 and the second imaging unit 12. In this case, itis possible to measure the distance to an object existing in the areawhere the imaging ranges of the first imaging unit 11 and the secondimaging unit 12 overlap with each other. Furthermore, in this case, whenthe stereo image processing is not performed, a wide-angle image imagedby the one to which the wide-angle lens is attached can be used for awide range of monitoring or the like, and an image imaged by the otherto which the telephoto lens or the like is attached can be used formonitoring a small object (for example, an object located at distant) orthe like.

The first correction unit 13 generates the basis image on the basis ofthe wide-angle image W1. In other words, as illustrated in FIG. 3, thefirst correction unit 13 projects the wide-angle image W1 supplied fromthe first imaging unit 11 onto the plane of the virtual sphericalsurface S at the projection angles 611, 612, and 613 to generate theplanar images P11, P12, and P13, and outputs the planar images P11, P12,and P13 to the first planar image pair processing unit 15, the secondplanar image pair processing unit 16, and the third planar image pairprocessing unit 17.

The second correction unit 14 generates the reference image on the basisof the wide-angle image W2. Note that, since an object existing on thebasis image is shifted on the reference image, the object existing in anend area of the basis image does not exist on the reference image, andit would happen that the distance to the object cannot be calculated. Toprevent such inconvenience, the reference image is generated with amargin with respect to the basis image.

FIG. 5 illustrates a relationship between the wide-angle image W2 andreference images P21, P22, and P23. As illustrated in FIG. 3, the secondcorrection unit 14 projects the wide-angle image W2 supplied from thesecond imaging unit 12 onto the plane of the virtual spherical surface Sat the projection angles θ21, θ22, and θ23 to generate the planar imagesP21, P22, and P23 having a margin M (P23 does not have a margin M), andoutputs the planar images P21, P22, and P23 to the first planar imagepair processing unit 15, the second planar image pair processing unit16, and the third planar image pair processing unit 17. Here, theprojection angle θ21 is an angle including the projection angle θ11 ofthe corresponding basis image. Similarly, the projection angle θ22 is anangle including the projection angle θ12 of the corresponding basisimage. Note that, in the case of FIG. 5, the margin M is not provided inthe planar image P23. The width and direction of the margin M may bedetermined depending on the search range of the corresponding points inthe stereo image processing. Note that, in a case where the base linelength of the first imaging unit 11 and the second imaging unit 12 islong, the width of the margin M needs to be made long accordingly.

The first planar image pair processing unit 15 performs the stereo imageprocessing using the planar image P11 as the basis image and the planarimage P21 as the reference image, measures the distances to the objectsexisting in the planar images P11 and P21, and outputs distanceinformation indicating the measurement result to the distanceinformation integration unit 18. Similarly, the second planar image pairprocessing unit 16 performs the stereo image processing using the planarimage P12 as the basis image and the planar image P22 as the referenceimage, measures the distances to the objects existing in the planarimages P12 and P22, and outputs distance information indicating themeasurement result to the distance information integration unit 18. Thethird planar image pair processing unit 17 performs the stereo imageprocessing using the planar image P13 as the basis image and the planarimage P23 as the reference image, measures the distances to the objectsexisting in the planar images P13 and P23, and outputs distanceinformation indicating the measurement result to the distanceinformation integration unit 18. As described above, the first planarimage pair processing unit 15, the second planar image pair processingunit 16, and the third planar image pair processing unit 17 do notdetect the corresponding points from the pair of the wide-angle imagesW, and detect the corresponding points from the pair of planar images Pwith corrected aberration. Therefore, the detection accuracy of thecorresponding points can be increased.

The distance information integration unit 18 integrates the distanceinformation input from the first planar image pair processing unit 15,the second planar image pair processing unit 16, and the third planarimage pair processing unit 17, and outputs the integrated distanceinformation to the distance information analysis unit 19. Morespecifically, the distance information integration unit 18 converts eachdistance information into a coordinate system based on one planar image(for example, the planar image P11) or into a polar coordinate systemcentered on the viewpoint of the first imaging unit 11 (or the secondimaging unit 12).

FIGS. 6 to 8 illustrate an example of integration of the distanceinformation. For example, as illustrated in FIG. 6, consider a case inwhich there is a wall 21 extending in a cross direction in front of theimaging direction. In this case, as illustrated in FIG. 7, the distanceof a wall 21-1 measured from the planar image pair P11 and P21 is mostdistant at a central portion of a screen and becomes closer toward theright side on the screen. The distance of a wall 21-2 measured from theplanar image pair P12 and P22 is constant. The distance of a wall 21-3measured from the planar image pair P13 and P23 is most distant at acentral portion of the screen and becomes closer toward the left side onthe screen.

The distance information integration unit 18 integrates the individualdistance information as illustrated in FIG. 7 and obtains the distanceto the entire wall 21 from the first imaging unit 11 and the secondimaging unit 12, as illustrated in FIG. 8.

Description refers back to FIG. 4. The distance information analysisunit 19 analyzes the integrated distance information to convert theintegrated distance information into information in a format suitablefor processing at a subsequent stage and outputs the information to thesubsequent stage. This output is used, for example, to realize variousfunctions such as approach warning, collision avoidance brake, lanechange warning, and automatic steering in the automobile.

<Distance Measurement Processing by Image Processing Device 10>

FIG. 5 is a flowchart for describing distance measurement processing bythe image processing device 10.

In step S1, the first correction unit 13 acquires the wide-angle imageW1 from the first imaging unit 11. Similarly, the second correction unit14 acquires the wide-angle image W2 from the second imaging unit 12.

In step S2, the first correction unit 13 generates the planar imagesP11, P12, and P13 that serve as the basis images for the stereo imageprocessing on the basis of the wide-angle image W1, and outputs theplanar images P11, P12, and P13 to the first planar image pairprocessing unit 15, the second planar image pair processing unit 16, andthe third planar image pair processing unit 17. Similarly, the secondcorrection unit 14 generates the planar images P21, P22, and P23 thatserve as the basis images for the stereo image processing on the basisof the wide-angle image W2, and outputs the planar images P21, P22, andP23 to the first planar image pair processing unit 15, the second planarimage pair processing unit 16, and the third planar image pairprocessing unit 17.

In step S3, the first planar image pair processing unit 15 performs thestereo image processing using the planar image P11 as the basis imageand the planar image P21 as the reference image, measures the distancesto the objects existing in the planar images P11 and P21, and outputsdistance information indicating the measurement result to the distanceinformation integration unit 18. Similarly, the second planar image pairprocessing unit 16 performs the stereo image processing using the planarimage P12 as the basis image and the planar image P22 as the referenceimage, measures the distances to the objects existing in the planarimages P12 and P22, and outputs distance information indicating themeasurement result to the distance information integration unit 18. Thethird planar image pair processing unit 17 performs the stereo imageprocessing using the planar image P13 as the basis image and the planarimage P23 as the reference image, measures the distances to the objectsexisting in the planar images P13 and P23, and outputs distanceinformation indicating the measurement result to the distanceinformation integration unit 18.

In step S4, the distance information integration unit 18 integrates thedistance information input from the first planar image pair processingunit 15, the second planar image pair processing unit 16, and the thirdplanar image pair processing unit 17, and outputs the integrateddistance information to the distance information analysis unit 19.

In step S5, the distance information analysis unit 19 analyzes theintegrated distance information to convert the integrated distanceinformation into information in a format suitable for processing at asubsequent stage and outputs the information to the subsequent stage.The description of the distance measurement processing is completed.

According to the above-described distance measurement processing, thedistance of an object in a wide range on an imaged wide-angle image canbe measured.

By the way, in the drawings used in the above description, the virtualspherical surface S at the time of capturing the wide-angle image W hasbeen illustrated in two dimensions. However, since the actual virtualspherical surface S is three-dimensional, the virtual spherical surfaceS may be divided into tile-like planes in order to divide the virtualspherical surface S into planes. However, although the three-dimensionalvirtual spherical surface S may be equally divided with respect to thespherical surface, in that case, division is difficult because eachplanar image cannot have a rectangular shape.

Therefore, in a case where it is not necessary to obtain the distance ofthe object at every angle of the virtual spherical surface S of when thewide-angle image W is imaged, it is a wide angle (about 180 degrees) ina specific plane direction, and it is sufficient to measure the distanceof the object existing in a range of a narrower angle than the wideangle in a coordinate axis direction orthogonal to the specific planedirection, the planar image W may be divided in a strip manner, asillustrated in FIG. 10. In the case of FIG. 10, it is the wide angle inan XZ plane direction, and the object existing in the range of anarrower angle than the wide angle can be measured in the Y-axisdirection orthogonal to the XZ plane direction.

FIG. 11 illustrates an example of arrangement of the first imaging unit11 and the second imaging unit 12 in a case where the planar image W isdivided in a strip manner, as illustrated in FIG. 10.

As illustrated in FIG. 11, if the first imaging unit 11 and the secondimaging unit 12 are arranged such that a direction of a line connectingthe first imaging unit 11 and the second imaging unit 12 (Y direction)and a division direction (X direction) become perpendicular to eachother, a search direction 31 of the corresponding points in the stereoimage processing coincides with the strip long-side direction (Ydirection). In this case, as illustrated in FIG. 5, the margin M is notnecessary in the reference image. However, strictly speaking, in a caseof performing block matching in the stereo image processing, a marginneeds to be provided in the reference image by a width corresponding tohalf the size of a block to be used.

<Case of Realizing Image Processing Device 10 by Program>

By the way, the above-described series of processing of the imageprocessing device 10 can be executed by hardware or by software. In thecase of executing the series of processing by software, a program thatconfigures the software is installed in a computer. Here, examples ofthe computer include a computer incorporated in dedicated hardware, anda general-purpose personal computer or the like capable of executingvarious functions by installing various programs, for example.

FIG. 12 is a block diagram illustrating a configuration example ofhardware of a computer that executes the above-described series ofprocessing by a program.

In a computer 100, a central processing unit (CPU) 101, a read onlymemory (ROM) 102, and a random access memory (RAM) 103 are mutuallyconnected by a bus 104.

Moreover, an input/output interface 105 is connected to the bus 104. Aninput unit 106, an output unit 107, a storage unit 108, a communicationunit 109, and a driver 110 are connected to the input/output interface105.

The input unit 106 includes a keyboard, a mouse, a microphone, and thelike. The output unit 107 includes a display, a speaker, and the like.The storage unit 108 includes a hard disk, a nonvolatile memory, and thelike. The communication unit 109 includes a network interface and thelike. The drive 110 drives a removable medium 111 such as a magneticdisk, an optical disk, a magneto-optical disk, or a semiconductormemory.

In the computer 100 configured as described above, the CPU 101, forexample, loads a program stored in the storage unit 108 into the RAM 103and executes the program via the input/output interface 105 and the bus104, whereby the above-described series of processing is performed.

The program to be executed by the computer 100 (CPU 201) can be recordedon the removable medium 111 as a package medium and the like, forexample, and provided. Furthermore, the program can be provided via awired or wireless transmission medium such as a local area network, theInternet, or digital broadcast.

Note that the program executed by the computer may be a programprocessed in chronological order according to the order described in thepresent specification or may be a program executed in parallel or atnecessary timing such as when a call is made.

<Case of Mounting Image Processing Device 10 on Vehicle>

Next, arrangement of the first imaging unit 11 and the second imagingunit 12 in the case of mounting the image processing device 10 on avehicle will be described.

FIG. 13 illustrates an arrangement example in the vehicle 511 of astereo camera 521 including the first imaging unit 11 and the secondimaging unit 12.

As illustrated in FIG. 13, the stereo camera 521 including the firstimaging unit 11 and the second imaging unit 12 can be installed in adoor mirror 512, a door mirror 513, a front frontal area, and a rearcenter of the vehicle 511.

A stereo camera 521A is installed at the door mirror 512 of the vehicle511. A stereo camera 521B is installed at the door mirror 513 of thevehicle 511. A stereo camera 521C is installed at the front frontal areaof the vehicle 511. A stereo camera 521D is installed at the rear centerof the vehicle 511.

As described above, the four directions of the vehicle 511 are monitoredby the stereo camera 521. At least one of the directions can bemonitored by an ultrasonic wave, a radar, a laser sensor, an infraredsensor, or the like. Moreover, a viewing system can be used incombination, in addition to obstacle recognition and monitoring by thestereo camera 521.

Next, FIGS. 14 and 15 illustrate an arrangement example of the firstimaging unit 11 and the second imaging unit 12 constituting the stereocamera 521 installed on the vehicle 511.

Although the first imaging unit 11 and the second imaging unit 12constituting the stereo camera 521 can be arranged in a lateraldirection, the first imaging unit 11 and the second imaging unit 12 canalso be arranged shifted up and down (in the vertical direction). Inaddition, the first imaging unit 11 and the second imaging unit 12 maybe arranged to have optical axes directed downward with respect to adirection parallel to a basis plane.

As illustrated in FIGS. 14 and 15, the first imaging unit 11 and thesecond imaging unit 12 constituting the stereo camera 521 are arrangedin an up-down direction (that is, the vertical direction) on a sidesurface of the vehicle 511. That is, the first imaging unit 11 and thesecond imaging unit 12 are arranged in a plane 553 perpendicular to abasis plane (road surface 551) to have a parallax in a height direction.Although attaching positions of the first imaging unit 11 and the secondimaging unit 12 are favorably a vicinity near a center of the sidesurface of the vehicle 511, there are some cases where installation isdifficult because there are a door and the like in the vicinity of thecenter. FIGS. 14 and 15 illustrate an example in which the first imagingunit 11 and the second imaging unit 12 are attached to the vicinity ofeach of door mirrors 512 and 513.

Furthermore, the reason for mounting the stereo camera 521 to thevicinity of each of the door mirrors 512 and 513 is that, as illustratedin FIG. 15, the stereo camera 521 can be attached obliquely downwardwithout adding a special jig in a case of attaching the stereo camera521 to be directed obliquely downward.

Note that, in FIGS. 14 and 15, the stereo camera 521 is installed onlyon the left side of the vehicle 511. However, in reality, the stereocamera 521 is installed on a right side as well.

Of course, the stereo camera 521 can be attached to a pillar (a frontpillar, a center pillar, a rear pillar, or the like), a door, a roofrail, or the like, other than to the door mirrors 512 and 513. Thestereo camera 521 may be attached to anywhere on the side surface of thevehicle 511.

Next, FIGS. 16, 17A, 17B, 17C, 18A, 18B, and 19 illustrate anotherarrangement example of the first imaging unit 11 and the second imagingunit 12 constituting the stereo camera 521 installed on the vehicle 511.

In the arrangement examples of FIGS. 14 and 15, the first imaging unit11 and the second imaging unit 12 are arranged on the side surfaces ofthe vehicle body of the vehicle 511 (specifically, on the door mirrors512 and 513) as illustrated in FIG. 16. In other words, as seen from thefront of the vehicle 511 (in the left diagram in FIG. 16), the secondimaging unit 12 is arranged above and the first imaging unit 11 isarranged below.

Then, the second imaging unit 12 is arranged at a position more distantfrom the vehicle 511 than the first imaging unit 11 (a position on anouter side of the vehicle 511), and the first imaging unit 11 isarranged at a position closer to the vehicle 511 than the second imagingunit 12 (a position on an inner side of the vehicle 511). A line 552connecting centers of the first imaging unit 11 and the second imagingunit 12 is inclined to jump out from the vehicle body to a monitoringdirection (to jump out from the side of the vehicle 511). In otherwords, the line 552 is inclined to jump out from a mounting surface(side surface) of the vehicle body of the vehicle 511. The stereo camera521 is not parallel to the vehicle body and is not perpendicular to theroad surface 551.

As seen from the front of the vehicle 511 (in the left diagram in FIG.16), both the first imaging unit 11 and the second imaging unit 12 aredirected in an obliquely downward direction of the vehicle 511. In otherwords, the first imaging unit 11 and the second imaging unit 12 areinclined in a plane including their optical axes 110 a and 120 a suchthat the optical axes 110 a and 120 a are directed downward with respectto a direction parallel to the basis plane (road surface 551) andintersect with the basis plane. That is, the first imaging unit 11 andthe second imaging unit 12 are inclined such that the line 552connecting the centers of the first imaging unit 11 and the secondimaging unit 12 forms an angle β, with respect to the basis plane. Inother words, the first imaging unit 11 and the second imaging unit 12are inclined such that their optical axes 110 a and 120 a are at anangle β, with respect to a line 553 perpendicular to the basis plane.

Furthermore, as seen from a top surface of the vehicle 511 (in the rightdiagram in FIG. 16), the optical axes 110 a and 120 a of the firstimaging unit 11 and the second imaging unit 12 are directed in adirection perpendicular to a traveling direction (downward in FIG. 16)of the vehicle 511, that is, in a direction perpendicular to the sidesurface of the vehicle 511.

In contrast, in the example illustrated in FIG. 17A, as seen from thefront of the vehicle 511 (in the left diagram in FIG. 17A), the secondimaging unit 12 is arranged above and the first imaging unit 11 isarranged below. Then, the first imaging unit 11 and the second imagingunit 12 are arranged at the same distance from the vehicle 511. In otherwords, the first imaging unit 11 and the second imaging unit 12 arearranged such that the line 552 connecting the centers of the firstimaging unit 11 and the second imaging unit 12 becomes parallel to thevehicle body (becomes perpendicular to the road surface 551 as the basisplane).

However, the first imaging unit 11 and the second imaging unit 12 areinclined in a plane including the optical axes 110 a and 120 a such thatthe optical axes 110 a and 120 a are directed downward with respect tothe direction parallel to the basis plane and intersect with the basisplane.

Furthermore, both the optical axes 11 oa and 12 oa of the first imagingunit 11 and the second imaging unit 12 are directed, as seen from thetop surface of the vehicle 511 (in the right diagram in FIG. 17A), in adirection perpendicular to the traveling direction (downward in FIGS.17A, 17B, and 17C) of the vehicle 511, that is, in a directionperpendicular to the side surface of the vehicle 511.

The configuration as seen from the front of the vehicle 511 of theexample illustrated in FIG. 17B (in the left diagram in FIG. 17B) issimilar to the case illustrated in the left diagram in FIG. 16.Repetitive description is omitted.

The configuration in the right diagram in FIG. 17B is different from theconfiguration in the right diagram in FIG. 16. In other words, in thisexample, both the optical axes 11 oa and 12 oa of the first imaging unit11 and the second imaging unit 12 are directed, as seen from the topsurface of the vehicle 511, slightly in the traveling direction, insteadof the direction perpendicular to the traveling direction (downward inFIGS. 17A, 17B, and 17C) of the vehicle 511. When the optical axes 11 oaand 12 oa are slightly directed in the traveling direction like this, itis advantageous to perform a distance measuring operation in cooperationwith the stereo camera 521 (for example, the stereo camera 521C formeasuring the distance in the range 522C in FIG. 13) used for measuringthe distance in the range in the traveling direction.

The configuration as seen from the front of the vehicle 511 of theexample illustrated in FIG. 17C (in the left diagram in FIG. 17C) issimilar to the case illustrated in the left diagram in FIG. 16.Repetitive description is omitted.

The configuration in the right diagram in C in FIG. 17C is differentfrom the configuration in the right diagram in FIG. 16. In other words,as seen from the top surface of the vehicle 511 (in the right diagram inC in FIG. 17C), the optical axis 12 oa of the second imaging unit 12 isdirected in a direction perpendicular to the traveling direction(downward in FIGS. 17A, 17B, and 17C) of the vehicle 511, that is, in adirection perpendicular to the side surface of the vehicle 511. That is,as far as the second imaging unit 12 is concerned, the configuration issimilar to that of the case in FIG. 16.

In contrast, as for the first imaging unit 11, the optical axis 11 oa isslightly directed in the traveling direction instead of in the directionperpendicular to the traveling direction (downward in FIGS. 17A, 17B,and 17C) of the vehicle 511. That is, as far as the first imaging unit11 is concerned, the configuration is similar to that of the case inFIG. 17B. Therefore, the relatively narrow hatched range in the diagramis the distance-measurable range as the stereo camera system. In a casewhere the distance-measurable range needs to be expanded, a camera withthe angle of view of 180 degrees or more can be used.

In the example illustrated in FIG. 18A, as seen from the front of thevehicle 511 (in the left diagram in FIG. 18A), the second imaging unit12 is arranged above and the first imaging unit 11 is arranged below.Then, the first imaging unit 11 and the second imaging unit 12 arearranged at the same distance from the vehicle 511. In other words, thefirst imaging unit 11 and the second imaging unit 12 are arranged suchthat the line 552 connecting the centers of the first imaging unit 11and the second imaging unit 12 becomes parallel to the vehicle body(becomes perpendicular to the road surface 551 as the basis plane).

Then, the first imaging unit 11 is directed in an obliquely downwarddirection of the vehicle 511 as seen from the front of the vehicle 511(in the left diagram in FIGS. 18A and 18B). In other words, the firstimaging unit 11 is inclined in a plane including the optical axis 11 oasuch that the optical axis 11 oa is directed downward with respect tothe direction parallel to the basis plane and intersects with the basisplane. The first imaging unit 11 is inclined such that the optical axis11 oa is at an angle β with respect to the line 553 perpendicular to thebasis plane. That is, as far as the first imaging unit 11 is concerned,the configuration is similar to that of the case in FIG. 16.

However, the second imaging unit 12 is arranged such that the opticalaxis 12 oa is directed parallel to the basis plane. That is, only one(the first imaging unit 11 arranged below) of the first imaging unit 11and the second imaging unit 12 is arranged such that the optical axis 11oa is directed downward with respect to the direction parallel to theroad surface 551 that is the basis plane, and intersects with the roadsurface 551. Then, the other (the second imaging unit 12 arranged above)is arranged such that the optical axis 12 oa becomes parallel to thebasis plane. Even when the first imaging unit 11 and the second imagingunit 12 are attached in this way, the hatched range in the vicinity ofthe vehicle 511 in FIGS. 18A and 18B are the distance-measurable range.The range is a relatively narrow range. In a case where thedistance-measurable range needs to be expanded, a camera with the angleof view of 180 degrees or more can be used.

The configuration as seen from the front of the vehicle 511 of theexample illustrated in FIG. 18A (in the right diagram in FIG. 18A) issimilar to the case illustrated in the right diagram in FIG. 16. Inother words, the optical axes 11 oa and 12 oa of the first imaging unit11 and the second imaging unit 12 are directed in the directionperpendicular to the traveling direction (downward in FIGS. 18A and 18B)of the vehicle 511, that is, in the direction perpendicular to the sidesurface of the vehicle 511.

In the example illustrated in FIG. 18B, as seen from the front of thevehicle 511 (in the left diagram in FIG. 18B), the second imaging unit12 is arranged above and the first imaging unit 11 is arranged below.Then, the second imaging unit 12 is arranged at a position more distantfrom the vehicle 511 than the first imaging unit 11, and the firstimaging unit 11 is arranged at a position closer to the vehicle 511 thanthe second imaging unit 12. The line 552 connecting centers of the firstimaging unit 11 and the second imaging unit 12 is inclined to jump outfrom the vehicle body to the monitoring direction (to jump out from theside of the vehicle 511). That is, the first imaging unit 11 and thesecond imaging unit 12 are inclined such that the line 552 connectingthe centers of the first imaging unit 11 and the second imaging unit 12forms an angle β with respect to the basis plane.

Then, the first imaging unit 11 is inclined in the plane including theoptical axis 110 a such that the optical axis 110 a is directed downwardwith respect to the direction parallel to the basis plane and intersectswith the basis plane. That is, the first imaging unit 11 is inclinedsuch that the line 552 connecting the centers of the first imaging unit11 and the second imaging unit 12 forms an angle β with respect to thebasis plane. In other words, the first imaging unit 11 is inclined suchthat the optical axis 110 a forms an angle β with respect to the line553 perpendicular to the basis plane.

However, the second imaging unit 12 is arranged such that the opticalaxis 12 oa is directed parallel to the basis plane. That is, only one(the first imaging unit 11 arranged below) of the first imaging unit 11and the second imaging unit 12 is arranged such that the optical axis 11oa is directed downward with respect to the direction parallel to theroad surface 551 that is the basis plane, and intersects with the roadsurface 551. Then, the other (the second imaging unit 12 arranged above)is arranged such that the optical axis 12 oa becomes parallel to thebasis plane. Even when the first imaging unit 11 and the second imagingunit 12 are attached in this way, the hatched range in the vicinity ofthe vehicle 511 in FIGS. 18A and 18B are the distance-measurable range.The range is a relatively narrow range. In a case where thedistance-measurable range needs to be expanded, a camera with the angleof view of 180 degrees or more can be used.

The configuration as seen from the front of the vehicle 511 of theexample illustrated in FIG. 18B (in the right diagram in FIG. 18B) issimilar to the case illustrated in the right diagram in FIG. 16. Inother words, the optical axes 11 oa and 12 oa of the first imaging unit11 and the second imaging unit 12 are directed in the directionperpendicular to the traveling direction (downward in FIGS. 18A and 18B)of the vehicle 511, that is, in the direction perpendicular to the sidesurface of the vehicle 511.

Note that various modifications may exist in the present technologywithin the scope not deviating from the essence of the presenttechnology.

Application Example

The technology according to the present disclosure can be applied tovarious products. For example, the technology according to the presentdisclosure may be realized as a device mounted on any type of vehiclessuch as an automobile, an electric automobile, a hybrid electricautomobile, an electric motorcycle, or the like.

FIG. 19 is a block diagram illustrating a schematic configurationexample of a vehicle control system 2000 to which the technology of thepresent disclosure is applicable. The vehicle control system 2000includes a plurality of electronic control units connected via acommunication network 2010. In the example illustrated in FIG. 19, thevehicle control system 2000 includes a drive system control unit 2100, abody system control unit 2200, a battery control unit 2300, a vehicleexterior information detection device 2400, a vehicle interiorinformation detection device 2500, and an integration control unit 2600.The communication network 2010 that connects the plurality of controlunits may be, for example, an on-board communication network conformingto an arbitrary standard such as a controller area network (CAN), alocal interconnect network (LIN), a local area network (LAN), or FlexRay(registered trademark), or a network conforming to a locally definedcommunication standard.

Each control unit includes, for example, a microcomputer that performsarithmetic processing according to various programs, a storage unit thatstores programs executed by the microcomputer, parameters used forvarious calculations, and the like, and a drive circuit that drivesvarious devices to be controlled. Each control unit includes a networkI/F for communicating with another control unit via the communicationnetwork 2010 and a communication I/F for communicating with a device, asensor, or the like outside the vehicle by wired communication orwireless communication. FIG. 19 illustrates, as functionalconfigurations of the integration control unit 2600, a microcomputer2610, a general-purpose communication I/F 2620, a dedicatedcommunication I/F 2630, a positioning unit 2640, a beacon reception unit2650, an in-vehicle device I/F 2660, an audio image output unit 2670, anon-board network I/F 2680, and a storage unit 2690. Similarly, the othercontrol units include a microcomputer, a communication I/F, a storageunit, and the like.

The drive system control unit 2100 controls an operation of a deviceregarding a drive system of a vehicle according to various programs. Forexample, the drive system control unit 2100 functions as a controldevice of a drive force generation device for generating drive force ofthe vehicle, such as an internal combustion engine or a drive motor, adrive force transmission mechanism for transmitting drive force towheels, a steering mechanism that adjusts a steering angle of thevehicle, a braking device that generates braking force of the vehicleand the like. The drive system control unit 2100 may have a function asa control device of an antilock brake system (ABS), electronic stabilitycontrol (ESC), or the like.

The drive system control unit 2100 is connected with a vehicle statedetection unit 2110. The vehicle state detection unit 2110 includes, forexample, at least one of a gyro sensor for detecting angular velocity ofan axial rotational motion of a vehicle body, an acceleration sensor fordetecting acceleration of the vehicle, or a sensor for detecting anoperation amount of an accelerator pedal, an operation amount of a brakepedal, a steering angle of a steering wheel, an engine speed, rotationspeed of a wheel, or the like. The drive system control unit 2100performs arithmetic processing using a signal input from the vehiclestate detection unit 2110 and controls the internal combustion engine,the drive motor, an electric power steering device, a brake device, orthe like.

The body system control unit 2200 controls operations of various devicesequipped in the vehicle body according to various programs. For example,the body system control unit 2200 functions as a control device of akeyless entry system, a smart key system, an automatic window device,and various lamps such as head lamps, back lamps, brake lamps, turnsignals, and fog lamps. In this case, radio waves transmitted from amobile device substituted for a key or signals of various switches canbe input to the body system control unit 2200. The body system controlunit 2200 receives an input of the radio waves or the signals, andcontrols a door lock device, the automatic window device, the lamps, andthe like of the vehicle.

The battery control unit 2300 controls a secondary battery 2310 that isa power supply source of the drive motor according to various programs.For example, the battery control unit 2300 receives information such asa battery temperature, a battery output voltage, or a remaining capacityof the battery from a battery device including the secondary battery2310. The battery control unit 2300 performs arithmetic processing usingthese signals to control temperature adjustment of the secondary battery2310, a cooling device provided in the battery device, or the like.

The vehicle exterior information detection device 2400 detectsinformation of an outside of the vehicle having the vehicle controlsystem 2000 mounted. For example, at least one of an imaging unit 2410or a vehicle exterior information detection unit 2420 is connected tothe vehicle exterior information detection device 2400. The imaging unit2410 includes at least one of a time of flight (ToF) camera, a stereocamera, a monocular camera, an infrared camera, or another camera. Thevehicle exterior information detection unit 2420 includes, for example,an environmental sensor for detecting current weather or atmosphericphenomena or an ambient information detection sensor for detecting othervehicles, obstacles, pedestrians, and the like around the vehicleequipped with the vehicle control system 2000.

The environmental sensor may be, for example, at least one of a raindropsensor for detecting rainy weather, a fog sensor for detecting fog, asunshine sensor for detecting the degree of sunshine, or a snow sensorfor detecting snowfall. The ambient information detection sensor may beat least one of an ultrasonic sensor, a radar device, or a lightdetection and ranging, laser imaging detection and ranging (LIDAR)device. The imaging unit 2410 and the vehicle exterior informationdetection unit 2420 may be provided as independent sensors or devices,respectively, or may be provided as devices in which a plurality ofsensors or devices is integrated.

Here, FIG. 20 illustrates an example of installation positions of theimaging unit 2410 and the vehicle exterior information detection unit2420. Each of imaging units 2910, 2912, 2914, 2916, and 2918 is providedat least one of positions such as a front nose, side mirrors, a rearbumper, a back door, and an upper portion of a windshield in an interiorof a vehicle 2900, for example. The imaging unit 2910 provided at thefront nose and the imaging unit 2918 provided at the upper portion ofthe windshield in the interior of the vehicle mainly acquire frontimages of the vehicle 2900. The imaging units 2912 and 2914 provided atthe side mirrors mainly acquire side images of the vehicle 2900. Theimaging unit 2916 provided at the rear bumper or the back door mainlyacquires a rear image of the vehicle 2900. The imaging unit 2918provided at the upper portion of the windshield in the interior of thevehicle is mainly used for detecting a preceding vehicle, a pedestrian,an obstacle, a traffic signal, a traffic sign, a lane, or the like.

Note that FIG. 20 illustrates an example of capture ranges of theimaging units 2910, 2912, 2914, and 2916. An imaging range a indicatesan imaging range of the imaging unit 2910 provided at the front nose,imaging ranges b and c respectively indicate imaging ranges of theimaging units 2912 and 2914 provided at the side mirrors, and an imagingrange d indicates an imaging range of the imaging unit 2916 provided atthe rear bumper or the back door. For example, a bird's-eye view imageof the vehicle 2900 as viewed from above can be obtained bysuperimposing image data imaged in the imaging units 2910, 2912, 2914,and 2916.

Vehicle exterior information detection units 2920, 2922, 2924, 2926,2928, and 2930 provided at the front, rear, side, corner, and upperportion of the windshield in the interior of the vehicle 2900 may beultrasonic sensors or radar devices, for example. Vehicle exteriorinformation detection units 2920, 2926, and 2930 provided at the frontnose, the rear bumper, the back door, and the upper portion of thewindshield in the interior of the vehicle 2900 may be LIDAR devices, forexample. These vehicle exterior information detection units 2920 to 2930are mainly used for detecting a preceding vehicle, a pedestrian, anobstacle, and the like.

Referring back to FIG. 19, the description will be continued. Thevehicle exterior information detection device 2400 causes the imagingunit 2410 to image an image of the outside the vehicle, and receivesimaged image data. Furthermore, the vehicle exterior informationdetection device 2400 receives detection information from the connectedvehicle exterior information detection unit 2420. In a case where thevehicle exterior information detection unit 2420 is an ultrasonicsensor, a radar device, or an LIDAR device, the vehicle exteriorinformation detection device 2400 transmits ultrasonic waves,electromagnetic waves, or the like and receives information of receivedreflected waves. The vehicle exterior information detection device 2400may perform object detection processing or distance detection processingfor persons, vehicles, obstacles, signs, letters, or the like on a roadsurface on the basis of the received information. The vehicle exteriorinformation detection device 2400 may perform environment recognitionprocessing of recognizing rainfall, fog, a road surface condition, orthe like on the basis of the received information. The vehicle exteriorinformation detection device 2400 may calculate the distance to theobject outside the vehicle on the basis of the received information.

Furthermore, the vehicle exterior information detection device 2400 mayperform image recognition processing or distance detection processing ofrecognizing persons, vehicles, obstacles, signs, letters, or the like ona road surface on the basis of the received image data. The vehicleexterior information detection device 2400 may perform processing suchas distortion correction or alignment for the received image data andcombine the image data imaged by different imaging units 2410 togenerate a bird's-eye view image or a panoramic image. The vehicleexterior information detection device 2400 may perform viewpointconversion processing using the image data imaged by the differentimaging units 2410.

The vehicle interior information detection device 2500 detectsinformation of an inside of the vehicle. The vehicle interiorinformation detection device 2500 is detected with a driver statedetection unit 2510 that detects a state of a driver, for example. Thedriver state detection unit 2510 may include a camera for imaging thedriver, a biometric sensor for detecting biological information of thedriver, a microphone for collecting sounds in the interior of thevehicle, and the like. The biometric sensor is provided, for example, ona seating surface, a steering wheel, or the like, and detects thebiological information of an occupant sitting on a seat or the driverholding the steering wheel. The vehicle interior information detectiondevice 2500 may calculate the degree of fatigue or the degree ofconcentration of the driver or may determine whether or not the driverfalls asleep at the wheel on the basis of detection information inputfrom the driver state detection unit 2510. The vehicle interiorinformation detection device 2500 may perform processing such as noisecanceling processing for collected sound signals.

The integration control unit 2600 controls the overall operation in thevehicle control system 2000 according to various programs. Theintegration control unit 2600 is connected with an input unit 2800. Theinput unit 2800 is realized by, a device that can be operated and inputby an occupant, such as a touch panel, a button, a microphone, a switch,or a lever, for example. The input unit 2800 may be, for example, aremote control device using an infrared ray or another radio waves, ormay be an externally connected device such as a mobile phone or apersonal digital assistant (PDA) corresponding to the operation of thevehicle control system 2000. The input unit 2800 may be, for example, acamera, and in this case, the occupant can input information by gesture.Moreover, the input unit 2800 may include, for example, an input controlcircuit that generates an input signal on the basis of the informationinput by the occupant or the like using the above input unit 2800 andoutputs the input signal to the integration control unit 2600, and thelike. The occupant or the like inputs various data to and instructs thevehicle control system 2000 on a processing operation by operating theinput unit 2800.

The storage unit 2690 may include a random access memory (RAM) forstoring various programs executed by the microcomputer, and a read onlymemory (ROM) for storing various parameters, a calculation result, asensor value, or the like. Furthermore, the storage unit 2690 may berealized by a magnetic storage device such as a hard disc drive (HDD), asemiconductor storage device, an optical storage device, amagneto-optical storage device, or the like.

The general-purpose communication I/F 2620 is a general-purposecommunication I/F that mediates communication with various devicesexisting in an external environment 2750. The general-purposecommunication I/F 2620 may include, for example, a cellularcommunication protocol such a global system of mobile communications(GSM) (registered trademark), WiMAX, long term evolution (LTE), orLTE-advanced (LTE-A), or another wireless communication protocol such asa wireless LAN (also called Wi-Fi (registered trademark)). Thegeneral-purpose communication I/F 2620 may be connected to a device (forexample, an application server or a control server) existing on anexternal network (for example, the Internet, a cloud network, or acompany specific network) via a base station or an access point, forexample. Furthermore, the general-purpose communication I/F 2620 may beconnected with a terminal (for example, a terminal of a pedestrian or ashop, or a machine type communication (MTC) terminal) existing in thevicinity of the vehicle, using a peer to peer (P2P) technology, forexample.

The dedicated communication I/F 2630 is a communication I/F supporting acommunication protocol formulated for use in the vehicle. The dedicatedcommunication I/F 2630 may include, for example, a standard protocolsuch as a wireless access in vehicle environment (WAVE) that is acombination of a lower layer IEEE 802.11p and an upper layer IEEE 1609,or dedicated short range communications (DSRC). The dedicatedcommunication I/F 2630 typically performs V2X communication that is aconcept including one or more of vehicle to vehicle communication,vehicle to infrastructure communication, and vehicle to pedestriancommunication.

The positioning unit 2640 receives a global navigation satellite system(GNSS) signal from a GNSS satellite (for example, a global positioningsystem (GPS) signal from a GPS satellite) to execute positioning, andgenerates position information including the latitude, longitude, andaltitude of the vehicle, for example. Note that the positioning unit2640 may specify a current position by exchanging signals with awireless access point or may acquire the position information from aterminal such as a mobile phone, a PHS, or a smartphone having apositioning function.

The beacon reception unit 2650 receives, for example, a radio wave or anelectromagnetic wave transmitted from a wireless station or the likeinstalled on a road, and acquires information such as a currentposition, congestion, road closure, or required time. Note that thefunction of the beacon reception unit 2650 may be included in theabove-described dedicated communication I/F 2630.

The in-vehicle device I/F 2660 is a communication interface thatmediates connection between the microcomputer 2610 and various devicesexisting in the vehicle. The in-vehicle device I/F 2660 may establishwireless connection using a wireless communication protocol such as awireless LAN, Bluetooth (registered trademark), near field communication(NFC), or wireless USB (WUSB). Furthermore, the in-vehicle device I/F2660 may establish wired connection via a connection terminal (notillustrated) (and a cable if necessary). The in-vehicle device I/F 2660exchanges control signals or data signals with, for example, a mobiledevice or a wearable device possessed by the occupant, or an informationdevice carried in or attached to the vehicle.

The on-board network I/F 2680 is an interface that mediatescommunication between the microcomputer 2610 and the communicationnetwork 2010. The on-board network I/F 2680 transmits and receivessignals and the like according to a predetermined protocol supported bythe communication network 2010.

The microcomputer 2610 of the integration control unit 2600 controls thevehicle control system 2000 according to various programs on the basisof information acquired via at least one of the general-purposecommunication I/F 2620, the dedicated communication I/F 2630, thepositioning unit 2640, the beacon reception unit 2650, the in-vehicledevice I/F 2660, or the on-board network I/F 2680. For example, themicrocomputer 2610 may calculate a control target value of the driveforce generation device, the steering mechanism, or the brake device onthe basis of the acquired information of the interior and the exteriorof the vehicle, and output a control command to the drive system controlunit 2100. For example, the microcomputer 2610 may perform cooperativecontrol for the purpose of avoiding a collision of the vehicle oralleviating impact, tracking based on the distance between vehicles,vehicle speed maintained traveling, automatic driving, or the like.

The microcomputer 2610 may create local map information includingperipheral information of the current position of the vehicle on thebasis of information acquired via at least one of the general-purposecommunication I/F 2620, the dedicated communication I/F 2630, thepositioning unit 2640, the beacon reception unit 2650, the in-vehicledevice I/F 2660, or the on-board network I/F 2680. Furthermore, themicrocomputer 2610 may predict danger such as a collision of thevehicle, approach of a pedestrian or the like, or entry of thepedestrian or the like into a closed road on the basis of the acquiredinformation, and generate a warning signal. The warning signal may be,for example, a signal for generating a warning sound or for lighting awarning lamp.

The audio image output unit 2670 transmits an output signal of at leastone of a sound or an image to an output device that can visually andaurally notify the occupant of the vehicle or outside the vehicle ofinformation. In the example in FIG. 19, as the output device, an audiospeaker 2710, a display unit 2720, and an instrument panel 2730 areexemplarily illustrated. The display unit 2720 may include, for example,at least one of an on-board display or a head-up display. The displayunit 2720 may have an augmented reality (AR) display function. Theoutput device may be another device such as a headphone, a projector, ora lamp, other than these devices. In the case where the output device isa display device, the display device visually displays a result obtainedin various types of processing performed by the microcomputer 2610 orinformation received from another control unit, in various formats suchas a text, an image, a table, and a graph. Furthermore, in the casewhere the output device is an audio output device, the audio outputdevice converts an audio signal including reproduced audio data,acoustic data, and the like into an analog signal, and aurally outputsthe analog signal.

Note that, in the example illustrated in FIG. 19, at least two controlunits connected via the communication network 2010 may be integrated asone control unit. Alternatively, an individual control unit may beconfigured by a plurality of control units. Moreover, the vehiclecontrol system 2000 may include another control unit (not illustrated).Furthermore, in the above description, some or all of the functionscarried out by any one of the control units may be performed by anothercontrol unit. That is, predetermined arithmetic processing may beperformed by any of the control units as long as information istransmitted and received via the communication network 2010. Similarly,a sensor or a device connected to any of the control units may beconnected to another control unit, and a plurality of control units maytransmit and receive detection information to each other via thecommunication network 2010.

In the above-described vehicle control system 2000, the image processingdevice 10 illustrated in FIG. 4 can be applied to the integrationcontrol unit 2600 of the application example illustrated in FIG. 19.

Note that embodiments of the present technology are not limited to theabove-described embodiments, and various modifications can be madewithout departing from the gist of the present technology.

The present technology can also have the following configurations.

(1)

An image processing device including:

-   -   a first generation unit configured to acquire a first image, and        project the first image on a plurality of planes on a virtual        spherical surface according to projection angles obtained by        dividing a viewing angle at time of capturing the first image to        generate the plurality of basis planar images;    -   a second generation unit configured to acquire a second image        including an area where imaging range overlaps with an imaging        range of the first image, and project the second image on a        plurality of planes on a virtual spherical surface according to        projection angles obtained by dividing a viewing angle at time        of capturing the second image to generate the plurality of        reference planar images; and    -   a plurality of stereo image processing units configured to        perform stereo image processing using a corresponding image pair        of the plurality of generated basis planar images and the        plurality of generated reference planar images to generate        distance information indicating a distance to an object on the        basis image.

(2)

The image processing device according to (1), in which

-   -   at least one of the first image or the second image is an image        imaged by a wide-angle camera.

(3)

The image processing device according to (1) or (2), in which

-   -   the second generation unit generates the plurality of reference        planar images provided with a margin with respect to the        plurality of basis planar images generated by the first        generation unit.

(4)

The image processing device according to (3), in which

-   -   a width of the margin is determined on the basis of a base line        length between a first imaging unit that images the first image        and a second imaging unit that images the second image.

(5)

The image processing device according to any one of (1) to (4), in which

-   -   an arranging direction of the plurality of basis planar images        and the plurality of reference planar images is orthogonal to a        direction of a base line length between a first imaging unit        that images the first image and a second imaging unit that        images the second image.

(6)

The image processing device according to any one of (1) to (5), in which

-   -   an arranging direction of the plurality of basis planar images        and the plurality of reference planar images is orthogonal to a        search direction of a corresponding point in the stereo image        processing.

(7)

The image processing device according to any one of (1) to (6), furtherincluding:

-   -   a distance information integration unit configured to integrate        the plurality of pieces of generated distance information.

(8)

The image processing device according to (7), in which

-   -   the distance information integration unit converts a coordinate        system of the plurality of pieces of generated distance        information.

(9)

The image processing device according to any one of (1) to (8), furtherincluding:

-   -   a first imaging unit configured to image the first image; and    -   a second imaging unit configured to image the second image.

(10)

The image processing device according to (9), in which

-   -   at least one of the first imaging unit or the second imaging        unit includes a wide-angle camera.

(11)

The image processing device according to (9) or (10), in which

-   -   the first imaging unit and the second imaging unit are arranged        side by side in a horizontal direction.

(12)

The image processing device according to (9) or (10), in which

-   -   the first imaging unit and the second imaging unit are arranged        up and down in a vertical direction.

(13)

An image processing method of an image processing device, the methodincluding:

-   -   by the image processing device,    -   a first generation step of acquiring a first image, and        projecting the first image on a plurality of planes on a virtual        spherical surface according to projection angles obtained by        dividing a viewing angle at time of capturing the first image to        generate the plurality of basis planar images;    -   a second generation step of acquiring a second image including        an area where imaging range overlaps with an imaging range of        the first image, and projecting the second image on a plurality        of planes on a virtual spherical surface according to projection        angles obtained by dividing a viewing angle at time of capturing        the second image to generate the plurality of reference planar        images; and    -   a plurality of stereo image processing steps of performing        stereo image processing using a corresponding image pair of the        plurality of generated basis planar images and the plurality of        generated reference planar images to generate distance        information indicating a distance to an object on the basis        image.

(14)

A vehicle including:

-   -   a first imaging unit configured to image a first image;    -   a second imaging unit configured to image second image including        an area where an imaging range overlaps with an imaging range of        the first image;    -   a first generation unit configured to acquire the first image,        and project the first image on a plurality of planes on a        virtual spherical surface according to projection angles        obtained by dividing a viewing angle at time of capturing the        first image to generate the plurality of basis planar images;    -   a second generation unit configured to acquire the second image,        and project the second image on a plurality of planes on a        virtual spherical surface according to projection angles        obtained by dividing a viewing angle at time of capturing the        second image to generate the plurality of reference planar        images; and    -   plurality of stereo image processing units configured to perform        stereo image processing using a corresponding image pair of the        plurality of generated basis planar images and the plurality of        generated reference planar images to generate distance        information indicating a distance to an object on the basis        image.

REFERENCE SIGNS LIST

-   10 Image processing device-   11 First imaging unit-   12 Second imaging unit-   13 First correction unit-   14 Second correction unit-   15 First planar image pair processing unit-   16 Second planar image pair processing unit-   17 Third planar image pair processing unit-   18 Distance information integration unit-   19 Distance information analysis unit-   100 Computer-   101 CPU

The invention claimed is:
 1. An image processing device, comprising: a first camera configured to image a first image; a second camera configured to image a second image, wherein the first camera and the second camera are on a vehicle, and the first camera is aligned with the second camera in a vertical direction; and a central processing unit (CPU) configured to: acquire the first image from the first camera; divide, at a time at which the first image is imaged, a viewing angle of the first image to obtain a first plurality of projection angles; project the first image on a first plurality of planes on a first virtual spherical surface based on the first plurality of projection angles; generate a plurality of basis planar images based on the projection of the first image on the first plurality of planes; acquire the second image from the second camera, wherein the second image includes an area where an imaging range of the second image overlaps with an imaging range of the first image; divide, at a time at which the second image is imaged, a viewing angle of the second image to obtain a second plurality of projection angles; project the second image on a second plurality of planes on a second virtual spherical surface based on the second plurality of projection angles; generate a plurality of reference planar images based on the projection of the second image on the second plurality of planes; execute stereo image processing based on corresponding image pairs of the plurality of basis planar images and the plurality of reference planar images; and generate, based on the execution of the stereo image processing, a plurality of pieces of distance information indicating a distance to an object in the first image.
 2. The image processing device according to claim 1, wherein the CPU is further configured to generate the plurality of reference planar images provided with a margin with respect to the plurality of basis planar images.
 3. The image processing device according to claim 2, wherein a width of the margin is based on a base line length between the first camera and the second camera.
 4. The image processing device according to claim 1, wherein an arranging direction of the plurality of basis planar images and the plurality of reference planar images is orthogonal to a direction of a base line length between the first camera and the second camera.
 5. The image processing device according to claim 1, wherein an arranging direction of the plurality of basis planar images and the plurality of reference planar images is orthogonal to a search direction of a corresponding point in the stereo image processing.
 6. The image processing device according to claim 1, wherein the CPU is further configured to integrate the plurality of pieces of distance information.
 7. The image processing device according to claim 6, wherein the CPU is further configured to convert a coordinate system of the plurality of pieces of generated distance information.
 8. The image processing device according to claim 1, wherein at least one of the first camera or the second camera is a wide-angle camera.
 9. An image processing method, comprising: in an image processing device that includes a first camera, a second camera, and a central processing unit (CPU): imaging a first image by the first camera; imaging a second image by the second camera, wherein the first camera and the second camera are on a vehicle, and the first camera is aligned with the second camera in a vertical direction; acquiring, by the CPU, the first image from the first camera; dividing, by the CPU at a time at which the first image is imaged, a viewing angle of the first image to obtain a first plurality of projection angles; projecting, by the CPU, the first image on a first plurality of planes on a first virtual spherical surface based on the first plurality of projection angles; generating, by the CPU, a plurality of basis planar images based on the projection of the first image on the first plurality of planes; acquiring, by the CPU, the second image from the second camera, wherein the second image includes an area where an imaging range of the second image overlaps with an imaging range of the first image; dividing, by the CPU at a time at which the second image is imaged, a viewing angle of the second image to obtain a second plurality of projection angles; projecting, by the CPU, the second image on a second plurality of planes on a second virtual spherical surface based on the second plurality of projection angles; generating, by the CPU, a plurality of reference planar images based on the projection of the second image on the second plurality of planes; executing, by the CPU, stereo image processing based on corresponding image pairs of the plurality of basis planar images and the plurality of reference planar images; and generating, by the CPU based on the execution of the stereo image processing, a plurality of pieces of distance information indicating a distance to an object in the first image.
 10. A vehicle, comprising: a first camera configured to image a first image; a second camera configured to image a second image, wherein the second image includes an area where an imaging range of the second image overlaps with an imaging range of the first image, and the first camera is aligned with the second camera in a vertical direction; and a central processing unit (CPU) configured to: acquire the first image from the first camera; divide, at a time at which the first image is imaged, a viewing angle of the first image to obtain a first plurality of projection angles; project the first image on a first plurality of planes on a first virtual spherical surface based on the first plurality of projection angles; generate a plurality of basis planar images based on the projection of the first image on the first plurality of planes; acquire the second image from the second camera; divide, at a time at which the second image is imaged, a viewing angle of the second image to obtain a second plurality of projection angles; project the second image on a second plurality of planes on a second virtual spherical surface based on the second plurality of projection angles; generate a plurality of reference planar images based on the projection of the second image on the second plurality of planes; execute stereo image processing based on corresponding image pairs of the plurality of basis planar images and the plurality of reference planar images; and generate, based on the execution of the stereo image processing, a plurality of pieces of distance information indicating a distance to an object in the first image. 